Psychological responses to simulated displays of mismatched emotional expressions

Embodied agents are often designed with the ability to simulate human emotion. This paper investigates the psychological impact of simulated emotional expressions on computer users with a particular emphasis on how mismatched facial and audio expressions are perceived (e.g. a happy face with a concerned voice). In a within-subjects repeated measures experiment (N = 68), mismatched animations were perceived as more engaging, warm, concerned and happy when a happy or warm face was in the animation (as opposed to a neutral or concerned face) and when a happy or warm voice was in the animation (as opposed to a neutral or concerned voice). The results appear to follow cognitive dissonance theory as subjects attempted to make mismatched expressions consistent on both the visual and audio dimensions of animations, resulting in confused perceptions of the emotional expressions. Design implications for affective embodied agents are discussed and future research areas identified.